System and method for displaying pertinent data to supplement information in images provided from a mobile communication device using augmented reality

ABSTRACT

Methods and systems are provided for displaying pertinent data, using augmented reality, to supplement information in images provided from a mobile communication device. The images can be analyzed to find one or more matched objects, and it can be determined whether recognized target components from the matched objects match one or more known patterns. Pertinent data pertaining to the known patterns can be sent to the mobile communication device so that it can be displayed using augmented reality to supplement information in the images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.15/826,802, filed Nov. 30, 2017, the contents of which are herebyincorporated by reference.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally tomobile computing devices. More particularly, embodiments of the subjectmatter relate to methods and systems for displaying pertinent data,using augmented reality, to supplement information in images providedfrom a mobile communication device.

BACKGROUND

Users of mobile communication devices such as smartphones, tabletcomputers or laptop computers often want to find information thatdescribes or supplements other information about the environment thatthey are currently using their mobile device within. This is commonlydone by using a search engine to search for websites or other sources ofinformation, and then retrieve information from one or more externaldata sources.

In some cases, the process of searching for appropriate information canbe inconvenient and/or time consuming. For example, the retrieval anddisplay of information can require access to multiple systems and/orrequire the user to perform many manual steps in searching for andretrieving the appropriate information.

It would be desirable to help automate this retrieval process andautomatically provide the user with information that helps the userbetter understand something about, for example, a person, place or thingin their present environment. Furthermore, other desirable features andcharacteristics will become apparent from the subsequent detaileddescription and the appended claims, taken in conjunction with theaccompanying drawings and the foregoing technical field and background.The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 is a schematic block diagram of an example of a computingenvironment in which features of the disclosed embodiments can beimplemented in accordance with the disclosed embodiments.

FIG. 2 is a flowchart illustrating a method for displaying pertinentdata pertaining to components recognized in images acquired by animaging device using augmented reality to supplement information in theimages in accordance with the disclosed embodiments.

FIG. 3 is a flowchart illustrating a method for displaying pertinentdata pertaining to a known text pattern recognized in images acquired byan imaging device using augmented reality to supplement information inthe images acquired by the imaging device in accordance with thedisclosed embodiments.

FIG. 4 is a flowchart illustrating a method for displaying pertinentdata pertaining to a specific person recognized in images acquired by animaging device using augmented reality to supplement information in theimages acquired by the imaging device in accordance with the disclosedembodiments.

FIG. 5 is a flowchart illustrating a method for displaying supplementalinformation that relates to a specific apparatus recognized in imagesacquired by an imaging device using augmented reality to supplementinformation in the images acquired by the imaging device in accordancewith the disclosed embodiments.

FIG. 6 is a flowchart illustrating a method for displaying a path as itis traversed along with supplemental information that is encounteredalong that path while traversing that path using augmented reality inaccordance with the disclosed embodiments.

FIG. 7 is a flowchart illustrating a method for displaying an image of alocation along with supplemental information that indicates other hiddenfeatures associated with that location using augmented reality inaccordance with the disclosed embodiments.

FIG. 8 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system within which a set of instructions,for causing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

FIG. 9 is a block diagram of an example implementation of a mobiledevice of FIG. 1 in accordance with the disclosed embodiments.

DETAILED DESCRIPTION

Augmented reality can refer to the integration of digital informationwith the user's environment in real time. Augmented reality technologiescan be used, for example, to superimpose a computer-generated image on auser's view of the real world, thus providing a composite view. Unlikevirtual reality, which creates a totally artificial environment,augmented reality uses the existing environment and overlays newinformation on top of it.

The disclosed embodiments relate to methods and systems for retrievingsupplemental information and displaying the supplemental information atthe mobile computing device using augmented reality. The disclosedembodiments can leverage various augmented reality technologies todisplay pertinent data or supplemental information about what is beingobserved via a camera of a user system (e.g., a mobile device) at adisplay associated with the user system. The pertinent data orsupplemental information can be retrieved from data sources such asbackend databases, backend server systems, cloud computing platforms,targets identified by search engines (such as Google Images service orGoggle reverse image search feature of Google Images service), socialmedia platforms or services, and provided to the user system. Thedisclosed embodiments can simplify retrieval and display of informationwhich would otherwise require access to multiple systems and many manualsteps. In one embodiment, where recognition processing can be performedlocally at the device in parallel with image capture and otherprocessing, the disclosed methodologies can occur in near real-time(e.g., so that the user perceives a smooth view with no stuttering tothe display).

In one embodiment, methods and systems are provided for displayingpertinent data, using augmented reality, to supplement information inimages acquired by an imaging device of a mobile communication device.The images can be analyzed to find one or more matched objects, andrecognition can be performed on the matched objects to recognize targetcomponents. It can be determined whether the recognized targetcomponents match one or more known patterns. Based on the knownpatterns, pertinent data pertaining to the known patterns can beretrieved from one or more data sources, and displayed at a displayusing augmented reality to supplement information in the images acquiredby the imaging device.

FIG. 1 is a schematic block diagram of an example of a computingenvironment or system 10 in which features of the disclosed embodimentscan be implemented in accordance with the disclosed embodiments. Thesystem 10 includes a user system, which in the example illustrated inFIG. 1, is a mobile communication device 100 that can communicate via anetwork interface (not illustrated in FIG. 1) with various recognitionsystems and databases 160 and various data sources 170 over a network120.

The mobile communication device 100 includes a processor 102, an imagingdevice 106, and a display 110 along with other conventional componentsthat are part of the mobile communication device 100 such as networkinterfaces for communicating with remote server systems and databasesover a network 120. Other examples of components that can be part ofmobile communication device 100 will be described below with referenceto example implementations that are shown FIGS. 8 and 9. However, itshould be appreciated that the disclosed embodiments can be implementedin conjunction with other types of user systems, such as desktops,laptops, tablets, smartphones or other client devices, Google Glass™,and any other computing device implemented in an automobile, aircraft,television, or other business or consumer electronic device or system,including web clients.

The recognition systems and databases 160 can vary depending on theimplementation and can include, for example, text recognition systemsand databases, image recognition systems and databases, facialrecognition systems and databases, landmark recognition systems anddatabases, recognition systems and databases, and any other known typeof recognition systems and databases.

The data sources 170 can include various different types of data sourcesthat can be used to provide information and data that can be used tosupplement other information that is displayed and/or identified at thedisplay 110. The one or more data sources data sources 170 can includeany number of backend systems including server systems and databases,cloud-based computing platforms, search engines, targeted data sourcesidentified by search engines, social media platforms or services, opengovernment data, etc. A cloud-based computing platform can include anetwork interface that allows a user of a user system to establish acommunicative connection to the cloud-based computing platform over anetwork 120 such as the Internet or any type of network describedherein. The cloud-based computing platform includes an applicationplatform that can give user systems access to various applications anddatabase systems provided by the application platform via a cloud-baseduser interface. Examples of backend systems can include, for example, anon-premises exchange server, the system/servers used by a search engine(e.g., Google) to allow users to perform searches, the system/serverused to retrieve Wikipedia articles based on user input, etc. Eachbackend system can include one or more servers that work in conjunctionwith one or more databases and/or data processing components.

Each of the recognition systems and databases 160 and data sources 170can be implemented using any number of servers (or server systems) anddatabases, repositories or other data storage systems that provide dataand/or services to the user systems. Each of the recognition systems anddatabases 160 and data sources 170 can be implemented using physicaland/or virtual database server hardware or computer systems that areconfigured to communicate with user systems to perform the variousfunctions described herein.

Each of the recognition systems and databases 160 and data sources 170can operate with any sort of conventional processing hardware, such as aprocessor, memory, input/output features and the like. The processorsmay be implemented using any suitable processing system, such as one ormore processors, controllers, microprocessors, microcontrollers,processing cores and/or other computing resources spread across anynumber of distributed or integrated systems, including any number of“cloud-based” or other virtual systems. Memory represents anynon-transitory short or long-term storage or other computer-readablemedia capable of storing programming instructions for execution on theprocessor, including any sort of random access memory (RAM), read onlymemory (ROM), flash memory, magnetic or optical mass storage, and/or thelike. The computer-executable programming instructions, when read andexecuted by the servers and/or processors, cause the server and/orprocessor to create, generate, or otherwise facilitate providing dataand information as described herein. It should be noted that the memoryrepresents one suitable implementation of such computer-readable media,and alternatively or additionally, a server could receive and cooperatewith external computer-readable media that is realized as a portable ormobile component or platform, e.g., a portable hard drive, a USB flashdrive, an optical disc, or the like. The input/output features generallyrepresent the interface(s) to networks (e.g., to the network 120, or anyother local area, wide area or other network), mass storage, displaydevices, data entry devices and/or the like.

Still referring to FIG. 1, the data and services provided by therecognition systems and databases 160 and data sources 170 can beretrieved using any sort of personal computer, smartphone, mobiletelephone, tablet or other network-enabled user system on the network120. In an exemplary embodiment, the user system 110 includes a displaydevice, such as a monitor, screen, or another conventional electronicdisplay capable of graphically presenting data and/or informationretrieved from the recognition systems and databases 160 and datasources 170, as described in greater detail below.

As will be described below, the imaging device 106 of the mobilecommunication device 100 can acquire images that can be processed viathe processor 102 and the recognition systems and databases 160, andthen used to request data that is pertinent to the information acquiredby the imaging device 106. This pertinent data can then be displayedusing augmented reality at the display 110 to provide supplementalinformation to the user of the mobile communication device 100. As usedherein, the phrase “using augmented reality” when used in conjunctionwith the term display or displaying can mean “presenting supplementalinformation on a normal view that is typically presented on a graphicaluser interface to provide a composite view that includes thesupplemental information presented.” For instance, in one context, thephrase “using augmented reality” when used in conjunction with the termdisplay or displaying can mean “superimposing a computer-generated imageor information on a normal view presented on a graphical user interfaceto provide a composite view that includes the computer-generated imageor information in addition to the normal view that is presented.”

In one embodiment, the imaging device 106 can be, for example, a cameraof a mobile device that is configured to acquire images of thesurrounding environment. The processor 102 can analyze the imagesacquired by the imaging device 106 to find one or more matched objects,and then perform recognition processing on the matched objects torecognize target components. The processor 102 can determine whether therecognized target components match one or more known patterns, andretrieve, based on the known patterns, pertinent data pertaining to theknown patterns from the one or more data sources 170. The display 110can display a user interface (e.g., GUI) that is configured to presentthe pertinent data, using augmented reality, to provide additionalinformation that is relevant to the scene captured by the imaging device106. The pertinent data that is displayed can supplement information inthe images acquired by the imaging device 106. For instance, thepertinent data can describe other information about objects or peoplethat are identified in images. In some implementations, thissupplemental information may be overlaid on the images being requiredacquired via the imaging device 106. In other implementations, thesupplemental information can be displayed independently of the imagesthat are acquired via the imaging device 106.

For example, in one embodiment, the matched objects can include matchedobjects that resemble text. The processor 102 can perform textrecognition on the matched objects that resemble text to recognize text,and determine whether the recognized text matches one or more known textpatterns by communicating with one or more textual recognition systemsand databases 160. The one or more textual recognition systems anddatabases 160 can generate text recognition results identifying theknown text patterns that match the recognized text, and return them tothe mobile communication device 100. The processor 102 can thenretrieve, based on the known text patterns that match the recognizedtext, pertinent data pertaining to the known text patterns that matchthe recognized text from one or more of the data sources 170. The userinterface displayed at the display 110 can present, using augmentedreality, at least some of the pertinent data to supplement informationin the images acquired by the imaging device 106.

In another embodiment, the matched objects can include matched objectsthat resemble a person. The processor 102 can perform facial recognitionon the matched objects that resemble the person to recognize facialfeatures the person, and determine whether the recognized features matchone or more known facial patterns by communicating with one or morefacial recognition systems and databases 160. The one or more textualrecognition systems and databases 160 can generate facial recognitionresults identifying a specific person having a known facial pattern thatmatches the recognized features, and return them to the mobilecommunication device 100. The processor 102 can then retrieve pertinentdata pertaining to the specific person from the one or more data sources170. The user interface displayed at the display 110 can present, usingaugmented reality, at least some of the pertinent data pertaining to thespecific person to supplement information in the images acquired by theimaging device 106.

In another embodiment, the matched objects can include matched objectsthat resemble an apparatus and identifiable information in the images.The processor 102 can perform recognition on the matched objects thatresemble the apparatus and the identifiable information to recognize theapparatus and the identifiable information, and determine whether therecognized apparatus matches any known apparatus by communicating withone or more recognition systems and databases 160. The one or morerecognition systems and databases 160 can generate recognition resultsidentifying the known apparatus that matches the recognized apparatus asa specific apparatus, and return them to the mobile communication device100. The processor 102 can then retrieve, based on the identifiableinformation, supplemental information that relates to the specificapparatus from the one or more data sources 170. The user interface(displayed at the display 110) can present, using augmented reality, atleast some of the supplemental information that relates to the specificapparatus to supplement information in the images acquired by theimaging device 106.

In another embodiment, the matched objects can include matched objectsthat resemble potential landmarks in the images. The processor 102 canreceive an input that indicates a desired destination, performrecognition on the matched objects that resemble the potential landmarksin the images to recognize landmarks in the images, and determinewhether each of the recognized landmarks matches any known landmarks bycommunicating with one or more recognition systems and databases 160.The one or more recognition systems and databases 160 can generaterecognition results identifying specific landmarks that match any knownlandmarks and location of each specific landmark, and return them to themobile communication device 100. The processor 102 can then retrieve,based on locations of each specific landmark and the desireddestination, one or more paths between the recognized landmarks and thedesired destination from the one or more data sources 170, and alsoretrieve (from the one or more data sources 170) supplementalinformation that is encountered along each path that can be displayedwhile traversing that path. The user interface can present at thedisplay 110, using augmented reality, a path as it is traversed alongwith at least some of the supplemental information that is encounteredalong that path while traversing that path.

In another embodiment, the matched objects can include matched objectsare potentially indicative of a current location in the images. Theprocessor 102 can perform recognition based on the matched objects thatare potentially indicative of current location in the images torecognize target components of each object, and determine whether eachof the recognized target components match any known patterns bycommunicating with one or more recognition systems and databases 160.The one or more recognition systems and databases 160 can generaterecognition results identifying specific objects having targetcomponents determined to match any known patterns as being objectsindicative of current location, and return them to the mobilecommunication device 100. The processor 102 can then determine thecurrent location based on the specific objects, and receive a selectionof one or more filters to be applied for generating supplementalinformation. The processor 102 can then retrieve, from the one or moredata sources 170, supplemental information that is to be displayed. Whendetermining supplemental information at the one or more data sources170, the one or more data sources 170 can apply the one or more filtersto determine appropriate supplemental information. The supplementalinformation is determined based on the one or more filters and canindicate, for example, other hidden features associated with the currentlocation that are obstructed from view and not visible. The userinterface can present the supplemental information at the display 110using augmented reality.

In one embodiment, the user of the mobile device can specify, based onuser input, a sub-set of pertinent data or supplemental information thatis allowed to be displayed at the display of the mobile device. This canhelp address privacy concerns. Depending on the system accessed, theuser may specify what “public” information may be displayed.

FIGS. 2-7 are flow charts that illustrate examples of methods fordisplaying pertinent data pertaining to components recognized in imagesacquired by a camera using augmented reality to supplement informationin the images in accordance with the disclosed embodiments. With respectto FIGS. 2-7, the steps of each method shown are not necessarilylimiting. Steps can be added, omitted, and/or performed simultaneouslywithout departing from the scope of the appended claims. Each method mayinclude any number of additional or alternative tasks, and the tasksshown need not be performed in the illustrated order. Each method may beincorporated into a more comprehensive procedure or process havingadditional functionality not described in detail herein. Moreover, oneor more of the tasks shown could potentially be omitted from anembodiment of each method as long as the intended overall functionalityremains intact. Further, each method is computer-implemented in thatvarious tasks or steps that are performed in connection with each methodmay be performed by software, hardware, firmware, or any combinationthereof. For illustrative purposes, the following description of eachmethod may refer to elements mentioned above in connection with FIG. 1.In certain embodiments, some or all steps of this process, and/orsubstantially equivalent steps, are performed by execution ofprocessor-readable instructions stored or included on aprocessor-readable medium. For instance, in the description of FIGS. 2-7that follows, the mobile communication device 100 (and various elementsthereof), recognition systems and databases 160, and data sources datasources 170 can be described as performing various acts, tasks or steps,but it should be appreciated that this refers to processing system(s) ofthese entities executing instructions to perform those various acts,tasks or steps. Depending on the implementation, some of the processingsystem(s) can be centrally located, or distributed among a number ofserver systems that work together. Furthermore, in the description ofFIGS. 2-7, a particular example is described in which a user systemmobile communication device 100 performs certain actions by interactingwith other elements of the system 10.

FIG. 2 is a flowchart illustrating a method 200 for displaying pertinentdata pertaining to components recognized in images acquired by animaging device 106 using augmented reality to supplement information inthe images in accordance with the disclosed embodiments. It should beappreciated that the method 200 can be executed continuously to updatethe pertinent data that is displayed via the GUI at display 110 so thatthe pertinent data corresponds to the images as they are acquired by theimaging device 106.

The method 200 begins at 210, where the imaging device 106 of the mobilecommunication device 100 acquires images and provides the images to theprocessor 102. At 220, the analyzer module 103 processes (e.g.,analyzes) the images to find matched objects, and provides the matchedobjects to the recognition module 104. At 230, the recognition module104 performs pattern recognition on the matched objects to generatetarget components, and the target components are then provided to thepattern analysis module 105.

At 240, the pattern analysis module 105 processes the target componentsand performs a pattern analysis of the target components to identifypatterns, and then provide the identified patters to the componentrecognition system and database 160-1, which processes the identifiedpatterns to determine whether they match any known patterns that arestored, and for any of the identified patterns that do match knownpatterns, generates recognition results identifying those knownpatterns. The recognition results can then be returned to the patternanalysis module 105 where they can be used by the processor 102. In oneimplementation, steps 230 and 240 can be performed by a combinedrecognition and pattern analysis module. In another implementation, therecognition module 104 and the pattern analysis module 105 are separatemodules, and the pattern analysis module 105 is an application specificmodule that determines whether the identified patterns match any knownpatterns that are relevant to a specific application.

At 250, the processor 102 can retrieve, based on the known patterns,pertinent data pertaining to the known patterns from one or more datasources data sources 170. For example, the processor 102 can generaterequest messages that are sent to the various data sources 170 torequest data pertaining to the known patterns from the recognitionresults. The data sources 170 can then search for pertinent datapertaining to the known patterns, and generate response messages thatinclude the pertinent data.

At 260, the processor 102 can then process and/or filter the pertinentdata from the responses messages as appropriate, and then cause at leastsome of the pertinent data to be displayed (at 270) via a user interfaceat the display 110 using augmented reality. The pertinent data that isdisplayed using augmented reality supplements information in the imagesthat were acquired by the camera.

Several different implementations of the general concept that isillustrated in FIG. 2 will now be described with reference to FIGS. 3-7.FIG. 3-7 will be described with continued reference to FIG. 1.

FIG. 3 is a flowchart illustrating a method 300 for displaying pertinentdata pertaining to a known text pattern recognized in images acquired byan imaging device 106 using augmented reality to supplement informationin the images acquired by the imaging device 106 in accordance with thedisclosed embodiments. It should be appreciated that the method 300 canbe executed continuously to update the pertinent data that is displayedvia the GUI at display 110 so that the pertinent data corresponds to theimages as they are acquired by the imaging device 106.

The method 300 begins at 310, where the imaging device 106 of the mobilecommunication device 100 acquires images and provides the images to theprocessor 102. At 320, the analyzer module 103 processes (e.g.,analyzes) the images to find matched objects that resemble text, andprovides the matched objects to the recognition module 104.

At 330, the recognition module 104 performs text recognition on thematched objects to recognize text, and the recognized text is thenprovided to the pattern analysis module 105. At 340, the patternanalysis module 105 processes the recognized text and performs a patternanalysis of the recognized text to identify patterns, and then providesthe identified patterns to the textual recognition system and database160-2, which processes the identified patterns to determine whether theymatch any known text patterns that are stored, and for any of theidentified patterns that do match known text patterns, generates textrecognition results identifying those known text patterns that match therecognized text. The text recognition results can then be returned tothe pattern analysis module 105 where they can be used by the processor102. In one implementation, steps 330 and 340 can be performed by acombined recognition and pattern analysis module. In anotherimplementation, the recognition module 104 and the pattern analysismodule 105 are separate modules, and the pattern analysis module 105 isan application specific module that determines whether the recognizedtext matches any known text patterns and is relevant to a specificapplication.

At 350, the processor 102 can retrieve, based on the known text patternsthat match the recognized text, pertinent data pertaining to the knowntext patterns that match the recognized text from one or more datasources data sources 170. For example, the processor 102 can generaterequest messages that are sent to the various data sources 170 torequest data pertaining to the known, matching text patterns from therecognition results. For instance, in one example of a retrievalprocess, after having identified text matching a known pattern, arequest is made to one or more data sources (e.g., a server or servers)with the identified text and the known user (if applicable) to retrievefurther information. The data sources 170 can then search for pertinentdata pertaining to the known, matching text patterns, and generateresponse messages that include the pertinent data.

At 360, the processor 102 can then process and/or filter the pertinentdata from the responses messages as appropriate, and then cause at leastsome of the pertinent data to be displayed (at 370) via a user interfaceat the display 110 using augmented reality. The pertinent data that isdisplayed using augmented reality supplements information (e.g., such asthe known, matching text patterns) in the images that were acquired bythe camera. The pertinent data could be displayed, for example, in atwo-dimensional or three-dimensional space depending on theimplementation. For example, in one non-limiting use case, a known,matching text pattern could be text that indicates the location of adesk or office, and the pertinent data could be information thatindicates who occupies the desk or office, information about his/hertitle, contact information, company assets (e.g., laptop, phone,desktop) assigned to the occupant or location, or any other informationabout the location that is displayed using augmented reality. In anothernon-limiting use case, a known, matching text pattern could be text thatindicates a room number of a conference room, and the pertinent datacould be booking information for that room that is displayed usingaugmented reality.

FIG. 4 is a flowchart illustrating a method 400 for displaying pertinentdata pertaining to a specific person recognized in images acquired by animaging device 106 using augmented reality to supplement information inthe images acquired by the imaging device 106 in accordance with thedisclosed embodiments. It should be appreciated that the method 400 canbe executed continuously to update the pertinent data that is displayedvia the GUI at display 110 so that the pertinent data corresponds to theimages as they are acquired by the imaging device 106.

The method 400 begins at 410, where the imaging device 106 of the mobilecommunication device 100 acquires images and provides the images to theprocessor 102. At 420, the analyzer module 103 processes (e.g.,analyzes) the images to find matched objects that resemble person, andprovides the matched objects to the recognition module 104.

At 430, the recognition module 104 performs facial recognition on thematched objects that resemble a person to recognize facial features, andthe recognized facial features are then provided to the pattern analysismodule 105. At 440, the pattern analysis module 105 processes therecognized facial features and performs a pattern analysis of therecognized facial features to identify facial patterns, and thenprovides the identified facial patterns to the face recognition systemand database 160-3, which processes the identified facial patterns todetermine whether they match any known facial patterns of a specificperson that are stored, and for any of the identified facial patternsthat do match known facial patterns, generates facial recognitionresults identifying a specific person having a known facial pattern thatmatches the recognized features. The facial recognition results can thenbe returned to the pattern analysis module 105 where they can be used bythe processor 102. In one implementation, steps 430 and 440 can beperformed by a combined recognition and pattern analysis module. Inanother implementation, the recognition module 104 and the patternanalysis module 105 are separate modules, and the pattern analysismodule 105 is an application specific module that determines whether theidentified facial patterns match any known facial patterns for aspecific person that are stored and for a specific application.

At 450, the processor 102 can retrieve pertinent data pertaining to thespecific person from the one or more data sources data sources 170. Forexample, the processor 102 can generate request messages that are sentto the various data sources 170 to request data pertaining to thespecific person from the recognition results (e.g., the specific personhaving the known, matching facial patterns). The data sources 170 canthen search for pertinent data pertaining to the specific person, andgenerate response messages that include the pertinent data.

At 460, the processor 102 can then process and/or filter the pertinentdata from the responses messages as appropriate, and then cause at leastsome of the pertinent data to be displayed (at 470) via a user interfaceat the display 110, for example, using augmented reality. The pertinentdata that is displayed can provide supplemental information about orassociated with the specific person identified in the images that wereacquired by the camera.

For example, once the person in the image has been identified,additional data (e.g., CRM data) associated with that person can then belooked-up and presented via a user interface of the device. Forinstance, in one non-limiting use case, the specific person could be asalesperson, and the pertinent data could be contact information forthat salesperson or any other information about that salesperson that isdisplayed using augmented reality. In another non-limiting use case,specific person could be any person who has one or more social mediaprofiles, and the pertinent data could be information extracted fromtheir social media profiles that is displayed using augmented reality.

FIG. 5 is a flowchart illustrating a method 500 for displayingsupplemental information that relates to a specific apparatus recognizedin images acquired by an imaging device 106 using augmented reality tosupplement information in the images acquired by the imaging device 106in accordance with the disclosed embodiments. It should be appreciatedthat the method 500 can be executed continuously to update thesupplemental information that is displayed via the GUI at display 110 sothat the supplemental information corresponds to the images as they areacquired by the imaging device 106.

The method 500 begins at 510, where the imaging device 106 of the mobilecommunication device 100 acquires images and provides the images to theprocessor 102. At 520, the analyzer module 103 processes (e.g.,analyzes) the images to find (1) matched objects that resemble anapparatus (e.g., equipment, a device, hardware such as a desktop orlaptop computer, office phone, assets assigned to a person, etc.) and/or(2) other identifiable information (e.g., name or picture of the personthat the apparatus is assigned to or any other information thatidentifies something about the apparatus), and provides the matchedobjects to the recognition module 104. Alternatively, identifiableinformation could be scanned by the device (e.g., by scanning a QRcode).

At 530, the recognition module 104 performs recognition on the matchedobjects to recognize the apparatus and/or the other identifiableinformation, and the recognized apparatus and/or other identifiableinformation are then provided to the pattern analysis module 105. At540, the pattern analysis module 105 processes the recognized apparatusand performs a pattern analysis of the recognized apparatus to identifyknown patterns, and then provides the identified patterns to theapparatus recognition system and database 160-4, which processes theidentified patterns to determine whether the recognized apparatus matchany known patterns for a specific apparatus that are stored, and for anyof the identified patterns that do match known patterns for a specificapparatus, generates recognition results identifying the specificapparatus. For example, in one embodiment, the processor can compare therecognized apparatus to a pre-trained intelligent data model to identifya known apparatus as a specific apparatus. The recognition results canthen be returned to the pattern analysis module 105 where they can beused by the processor 102. In one implementation, steps 530 and 540 canbe performed by a combined recognition and pattern analysis module. Inanother implementation, the recognition module 104 and the patternanalysis module 105 are separate modules, and the pattern analysismodule 105 is an application specific module that determines whether therecognized apparatus matches any known patterns for a specific apparatusand are relevant to a specific application being used to identify thespecific apparatus.

In some embodiments, at 540, the pattern analysis module 105 alsoprocesses the recognized identifiable information and performs a patternanalysis of the recognized identifiable information to identify knownpatterns, and then provides the identified patterns to the aninformation recognition system and database (not shown), which processesthe identified patterns to determine whether they match any knownpatterns for information that are stored, and for any of the identifiedpatterns that do match known patterns for specific information,generates recognition results identifying the specific identifiableinformation. The recognition results can then be returned to the patternanalysis module 105 where they can be used by the processor 102

At 550, the processor 102 can retrieve, based on the identifiableinformation, supplemental information that relates to the specificapparatus from the one or more data sources data sources 170. Forexample, the processor 102 can use the identifiable information toretrieve supplemental information from data sources (e.g., supplementalinformation that identifies the location, owner of the space, and theapparatus allocated to the owner, etc.) In one embodiment, the processor102 can generate request messages that are sent to the various datasources 170 to request supplemental information that relates to thespecific apparatus and/or the specific identifiable information from therecognition results (e.g., supplemental information about the specificapparatus having the specific identifiable information). The datasources 170 can then search for the supplemental information, andgenerate response messages that include the supplemental information.

At 560, the processor 102 can then process and/or filter thesupplemental information from the responses messages as appropriate, andthen cause at least some of the supplemental information to be displayed(at 570) via a user interface at the display 110, for example, usingaugmented reality. The supplemental information that is displayed canprovide additional information about or associated with the specificapparatus identified in the images that were acquired by the camera.

For example, in one non-limiting use case, specific apparatus could becomputer monitor, and the other identifiable information could be abrand or model identifier. The supplemental information that isdisplayed using augmented reality could identify the monitor(s) that areassigned to a specific user.

In another non-limiting use case, a user who is part of an assetmanagement team could use their mobile device to acquire identifiableinformation that specifies an room, office or cube number, and aspecific piece of office equipment (e.g., a computer monitor) located inthe corresponding space. This identifiable information could then beused to retrieve supplemental information about the occupant of thatspace from a data source and display it along with other informationabout the office equipment located in that space. Here, the specificapparatus could be the office equipment or other assets located in thatspace, and the supplemental information about the occupant of that spacecould be information about the occupant and the office equipment orother assets that are assigned to that occupant. Some or all of thisinformation could be displayed using augmented reality. The assetmanagement team member could then use the information about the officeequipment that is actually located in the space and compare it to theoffice equipment that is assigned to that occupant to determine whatdoes belong and what does not. For instance, the supplementalinformation could include a list of assigned assets with a check markagainst those that are identified as being present and flags thatindicate which assets are missing.

FIG. 6 is a flowchart illustrating a method 600 for displaying a path asit is traversed along with supplemental information that is encounteredalong that path while traversing that path using augmented reality inaccordance with the disclosed embodiments. It should be appreciated thatthe method 600 can be executed continuously to update the supplementalinformation that is displayed via the GUI at display 110.

The method 600 begins at 605, where the processor 102 of the mobilecommunication device 100 determines a desired destination of a user. At610, the imaging device 106 acquires images and provides the images tothe processor 102. At 620, the analyzer module 103 processes (e.g.,analyzes) the images to find matched objects that resemble potentiallandmarks, and provides the matched objects to the recognition module104. As used herein, a “landmark” can refer to “an object or featurethat can be identified in an image and used to identify or establishlocation or relative location within an environment.” In some cases, alandmark can be used to establish orientation within a space, or toindicate directions for moving toward a destination.

At 630, the recognition module 104 performs landmark recognition on thematched objects that resemble potential landmarks to recognizelandmarks, and the recognized landmarks are then provided to the patternanalysis module 105. At 640, the pattern analysis module 105 processesthe recognized landmarks and performs a pattern analysis of therecognized landmarks to identify patterns, and then provides theidentified patterns to the landmark recognition system and database160-5, which processes the identified patterns to determine whether eachof the recognized landmarks match any known landmarks from specificlandmarks that are stored, and for any of the identified patterns thatdo match known landmarks, generates landmark recognition resultsidentifying one or more specific landmarks and respective location(s) ofthose specific landmarks. In one embodiment, the recognized landmarkscan be compared to a pre-trained intelligent data model to identify aknown landmarks as a specific landmarks. In one implementation, steps630 and 640 can be performed by a combined recognition and patternanalysis module. In another implementation, the recognition module 104and the pattern analysis module 105 are separate modules, and thepattern analysis module 105 is an application specific module thatdetermines whether each of the recognized landmarks match any knownspecific landmarks that are relevant to a specific application.

The landmark recognition results can then be returned to the patternanalysis module 105 where they can be used by the processor 102. At 645,the processor 102 can determine the locations of the recognizedlandmarks (from the landmark recognition results).

At 650, the processor 102 can retrieve, based on locations of eachspecific landmark and the desired destination, one or more paths betweenthe recognized landmarks and the desired destination from the one ormore data sources data sources 170, and also retrieve, from the one ormore data sources data sources 170, supplemental information that isencountered along each path that can be displayed while traversing thatpath. For example, the processor 102 can generate request messages thatare sent to one or more of the various data sources 170 to request apath between one or more of the recognized landmarks and the desireddestination, and other request messages to also request supplementalinformation that is encountered along each path that can be displayedwhile traversing that path. One or more of the data sources 170 cansearch for paths between one or more of the recognized landmarks and thedesired destination, and generate response messages that described oneor more available paths between one or more of the recognized landmarksand the desired destination. For example, in one embodiment, one or moreof the data sources 170 can search for supplemental information, andgenerate response messages that include the supplemental information.

At 660, as a user moves along or traverses one of the paths, theprocessor 102 can then process and/or filter the supplementalinformation from the responses messages as appropriate, and then causeat least some of the supplemental information to be displayed (at 670)via a user interface at the display 110, for example, using augmentedreality. In other words, as the path is traversed it can be displayed atthe display 110 along with at least some of the supplemental informationthat is encountered along that path while traversing that path, and thissupplemental information can be displayed using augmented reality. Thesupplemental information that is displayed can provide, for example,supplemental information about or associated with anything that isencountered along the path such as objects that are encountered. Forexample, in one non-limiting use case, the method 600 could be used by auser in large facility or office building to enter a desireddestination, and then acquire an image of their current location thatincludes a particular landmark. The method 600 could then be used todisplay a path between the current location and desired destination andalong with other supplemental information such as shortcuts, otheridentifiable landmarks, etc. In one non-limiting embodiment, thesupplemental information could as simple as arrows indicating thedirection the user must follow to reach the destination (e.g., like turnby turn directions provided by a navigation system), and could alsooptionally include the number of steps or the distance to thedestination to give the user a better idea of the distance.

FIG. 7 is a flowchart illustrating a method 700 for displaying an imageof a location along with supplemental information that indicates otherhidden features associated with that location using augmented reality inaccordance with the disclosed embodiments. It should be appreciated thatthe method 700 can be executed continuously to update the supplementaldata that is displayed via the GUI at display 110 so that thesupplemental data corresponds to the images as they are acquired by theimaging device 106. The supplemental information can indicate otherhidden features at a location that are obstructed from view and notvisible to the human eye. For instance, in one implementation, thehidden features can be located, for example, underneath the location.The hidden features that are displayed can vary depending on whichfilter within an augmented reality view a user has selected. Forinstance, one filter would allow the user to view structure underlying aparticular location. For a building with geo-location, the user could besee an augmented reality view of girders, water pipes, electricalsystem, etc. By contrast, if the camera of the device is pointed at theroad, the user could be see an augmented reality view of the subway,sewer lines, water pipes, etc.

The method 700 begins at 710, where the imaging device 106 of the mobilecommunication device 100 acquires images and provides the images to theprocessor 102.

At 720, the analyzer module 103 processes (e.g., analyzes) the images tofind matched objects or information that are potentially indicative ofcurrent location in the images, and provides the matched objects thatare potentially indicative of current location in the images to therecognition module 104.

At 730, the recognition module 104 performs pattern recognition on thematched objects to generate target components, and the target componentsare then provided to the pattern analysis module 105. For example, therecognition module 104 can perform recognition based on the matchedobjects that are potentially indicative of current location in theimages to recognize target components of each object.

At 740, the pattern analysis module 105 processes the target componentsand performs a pattern analysis of the target components to identifypatterns, and then provides the identified patterns to the componentrecognition system and database 160-1. The component recognition systemand database 160-1 processes the identified patterns to determinewhether they match any known patterns that are stored, and for any ofthe identified patterns that do match known patterns, generatesrecognition results identifying those known patterns. For instance, inone embodiment, the recognized target components can be compared to apre-trained intelligent data model to identify specific objects havingtarget components determined to match any known patterns, and identifythose specific objects as being indicative of current location. In oneimplementation, steps 730 and 740 can be performed by a combinedrecognition and pattern analysis module. In another implementation, therecognition module 104 and the pattern analysis module 105 are separatemodules, and the pattern analysis module 105 is an application specificmodule that determines whether the identified patterns match any knownpatterns that are relevant to a specific application.

The recognition results can then be returned to the pattern analysismodule 105 where they can be used by the processor 102. At 742, thecurrent location shown in the images can be determined based on specificobjects that are determined to be indicative of the current location.For example, in one embodiment, the processor 102 can determine thecurrent location based on the specific objects by providing the specificobjects to a location determination system and database imaging device106-6 that analyzes the specific objects to find matching objects withina database. Each matching object can have a specific location associatedwith it.

At 745, the processor 102 can receive a selection of one or more filtersto be applied for generating supplemental information. The selection canbe based on a user input or selection, or can be automated based oninformation determined by the device, for example, based on informationincluded in the images that are acquired by the camera.

At 750, supplemental information that is to be displayed is retrievedfrom the one or more data sources data sources 170. One or more filtersare applied at the one or more data sources data sources 170 todetermine the supplemental information based on the one or more filters.For example, in one implementation, the processor 102 can generaterequest messages that are sent to one or more of the various datasources 170 to request supplemental information, and the data sources170 can then apply the filter(s) and search for the supplementalinformation. The data sources 170 can then generate response messagesthat include the supplemental information, and send the responsemessages to the processor 102.

At 760, the processor 102 can then process supplemental information fromthe responses messages as appropriate, and then cause at least some ofthe supplemental information to be displayed (at 770) via a userinterface at the display 110 using augmented reality. The supplementalinformation that is displayed using augmented reality supplementsinformation in the images that were acquired by the camera and indicatesother hidden features associated with the current location that areobstructed from view and not visible (e.g., that are located, forexample, underneath the current location).

FIG. 8 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system 800 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. The system 800 may bein the form of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed. In alternative embodiments, themachine may be connected (e.g., networked) to other machines in a LAN,an intranet, an extranet, or the Internet. The machine may operate inthe capacity of a server machine in client-server network environment.The machine may be a personal computer (PC), a set-top box (STB), aserver, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein. In one embodiment, computer system800 may represent a user system 100, one of the recognition systems anddatabases 160 (or a components thereof) or one of the data sources 170(or a components thereof) as shown in FIG. 1.

The exemplary computer system 800 includes a processing device(processor) 802, a main memory 804 (e.g., read-only memory (ROM), flashmemory, dynamic random access memory (DRAM) such as synchronous DRAM(SDRAM)), a static memory 806 (e.g., flash memory, static random accessmemory (SRAM)), and a data storage device 818, which communicate witheach other via a bus 830.

Processing device 802 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device 802 may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or a processor implementing other instruction sets orprocessors implementing a combination of instruction sets. Theprocessing device 802 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like.

The computer system 800 may further include a network interface device808. The computer system 800 also may include a video display unit 810(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 812 (e.g., a keyboard), a cursor controldevice 814 (e.g., a mouse), and a signal generation device 816 (e.g., aspeaker).

The data storage device 818 may include a computer-readable medium 828on which is stored one or more sets of instructions 822 (e.g.,instructions of in-memory buffer service 114) embodying any one or moreof the methodologies or functions described herein. The instructions 822may also reside, completely or at least partially, within the mainmemory 804 and/or within processing logic 826 of the processing device802 during execution thereof by the computer system 800, the main memory804 and the processing device 802 also constituting computer-readablemedia. The instructions may further be transmitted or received over anetwork 820 via the network interface device 808.

While the computer-readable storage medium 828 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing, encoding or carrying a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present invention.The term “computer-readable storage medium” shall accordingly be takento include, but not be limited to, solid-state memories, optical media,and magnetic media.

FIG. 9 is a block diagram of an example implementation 900 of the mobilecommunication device 100 of FIG. 1. The mobile communication device 100can include a memory interface 902, one or more data processors, imageprocessors and/or central processing units 904, and a peripheralsinterface 906. The memory interface 902, the one or more processors 904and/or the peripherals interface 906 can be separate components or canbe integrated in one or more integrated circuits. The various componentsin the mobile communication device 100 can be coupled by one or morecommunication buses or signal lines.

Sensors, devices, and subsystems can be coupled to the peripheralsinterface 906 to facilitate multiple functionalities. For example, amotion sensor 910, a light sensor 912, and a proximity sensor 914 can becoupled to the peripherals interface 906 to facilitate the orientation,lighting, and proximity functions. A hardware connection sensor 918 canbe coupled to the peripherals interface 906, to facilitate determining astate of connecting the mobile communication device 100 to any hardware,e.g., a docking station, a charger, a personal computer, etc. A grippingsensor 919 can be coupled to the peripherals interface 906, to determineif the mobile communication device 100 is being gripped. In variousimplementation, a gripping sensor can include a temperature sensor,and/or a pressure sensor. Further, a touch sensor 921 can be coupled tothe peripherals interface 906, to detect if a user is touching userinput interface, e.g., a touch screen or a keypad. A time sensor 923 canalso be coupled to the peripherals interface 906, to detect a durationof a certain state of the mobile communication device 100. Other sensors916 can also be connected to the peripherals interface 906, such as apositioning system (e.g., GPS receiver), a temperature sensor, abiometric sensor, a gyroscope, or other sensing device, to facilitaterelated functionalities.

A camera subsystem 920 and an optical sensor 922, e.g., a chargedcoupled device CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor, can be utilized to facilitate camera functions, such asrecording photographs and video clips.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 924, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of thecommunication subsystem 924 can depend on the communication network(s)over which the mobile communication device 100 is intended to operate.In particular, the wireless communication subsystems 924 may includehosting protocols such that the device 100 may be configured as a basestation for other wireless devices.

An audio subsystem 926 can be coupled to a loudspeaker 124, andmicrophone 122 to facilitate voice-enabled functions, for example,hands-free functionalities, voice recognition, voice replication,digital recording, and telephony functions.

The I/O subsystem 940 can include a touch screen controller 942 and/orother input controller(s) 944. The touch-screen controller 942 can becoupled to a touch screen 946. The touch screen 946 and touch screencontroller 942 can, for example, detect contact and movement or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies, as well as other proximity sensorarrays or other elements for determining one or more points of contactwith the touch screen 946.

The other input controller(s) 944 can be coupled to other input; controldevices 948, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port; and/or a pointer device such as a stylus. Theone or more buttons (not shown) can include an up/down button for volumecontrol of the speaker 126 and loudspeaker 124 and/or the microphone122.

In some implementations, the mobile communication device 100 can presentrecorded audio and/or video files, such as MP3, AAC, and MPEG files. Insome implementations, the mobile communication device 100 can includethe functionality of an MP3 player, such as an iPod™. The mobilecommunication device 100 may, therefore, include a 96-pin connector thatis compatible with the iPod. Other input/output and control devices canalso be used.

The memory interface 902 can be coupled to memory 950. The memory 950can include high-speed random access memory and/or non-volatile memory,such as one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR). The memory 950can store an operating system 952, such as Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system such as VxWorks. Theoperating system 952 may include instructions for handling basic systemservices and for performing hardware dependent tasks. In someimplementations, the operating system 952 can be a kernel (e.g., UNIXkernel).

The memory 950 may also store communication instructions 954 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. The memory 950 may includegraphical user interface instructions 956 to facilitate graphic userinterface processing; sensor processing instructions 958 to facilitatesensor-related processing and functions; phone instructions 960 tofacilitate phone-related processes and functions; electronic messaginginstructions 962 to facilitate electronic-messaging related processesand functions; web browsing instructions 964 to facilitate webbrowsing-related processes and functions; media processing instructions966 to facilitate media processing-related processes and functions;GPS/navigation instructions 968 to facilitate GPS and navigation-relatedprocesses and instructions; camera instructions 970 to facilitatecamera-related processes and functions; GUI adjustment instructions 973to facilitate adjustment of graphical user interfaces and user interfaceelements in response to sensor data; and/or other software instructions972 to facilitate other processes and functions.

In addition, the memory 950 can store audio management instructions 976to facilitate functions managing audio subsystem, including theloudspeaker 124, and the microphone 122. In some implementations, theaudio management instructions 976 are operable to toggle thespeakerphone system and adjust speaker volume and/or microphonesensitivity, in response to the sensor processing instructions 958.

The memory 950 may also store other software instructions (not shown),such as web video instructions to facilitate web video-related processesand functions; and/or web shopping instructions to facilitate webshopping-related processes and functions. In some implementations, themedia processing instructions 966 are divided into audio processinginstructions and video processing instructions to facilitate audioprocessing-related processes and functions and video processing-relatedprocesses and functions, respectively. An activation record andInternational Mobile Equipment Identity (IMEI) 974 or similar hardwareidentifier can also be stored in memory 950.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. The memory 950 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the mobile communication device 100 may be implemented inhardware and/or in software, including in one or more signal processingand/or application specific integrated circuits.

The preceding description sets forth numerous specific details such asexamples of specific systems, components, methods, and so forth, inorder to provide a good understanding of several embodiments of thepresent invention. It will be apparent to one skilled in the art,however, that at least some embodiments of the present invention may bepracticed without these specific details. In other instances, well-knowncomponents or methods are not described in detail or are presented insimple block diagram format in order to avoid unnecessarily obscuringthe present invention. Thus, the specific details set forth are merelyexemplary. Particular implementations may vary from these exemplarydetails and still be contemplated to be within the scope of the presentinvention.

In the above description, numerous details are set forth. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that embodiments of the invention may bepracticed without these specific details. In some instances, well-knownstructures and devices are shown in block diagram form, rather than indetail, in order to avoid obscuring the description.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “determining,” “analyzing,” “identifying,” “adding,”“displaying,” “generating,” “querying,” “creating,” “selecting” or thelike, refer to the actions and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (e.g., electronic) quantities within thecomputer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. This apparatus may be specially constructed forthe required purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

What is claimed is:
 1. A method, comprising: analyzing images providedfrom a mobile communication device, at a processor, to find one or morematched objects; determining, at the processor, whether recognizedtarget components from the matched objects match one or more knownpatterns; and sending, from the processor to the mobile communicationdevice, pertinent data pertaining to the known patterns to supplementinformation in the images, wherein the pertinent data is to be displayedat a display of the mobile communication device using augmented reality.2. A method according to claim 1, further comprising: receiving imagesacquired by an imaging device of the mobile communication device;performing recognition on the matched objects, at the processor, torecognize target components; retrieving, based on the known patterns,the pertinent data pertaining to the known patterns from one or moredata sources; and displaying, at a display of the mobile communicationdevice using augmented reality, at least some of the pertinent data tosupplement the information in the images acquired by the imaging device.3. A method according to claim 2, wherein analyzing the images,comprises: analyzing the images, at the processor, to find the matchedobjects that resemble text; wherein performing recognition on thematched objects, at the processor, to recognize target components,comprises: performing text recognition on the matched objects thatresemble text, at the processor, to recognize text; wherein determiningcomprises: determining, at the processor, whether the recognized textmatches one or more known text patterns, and generating text recognitionresults identifying the known text patterns that match the recognizedtext; wherein retrieving comprises: retrieving, based on the known textpatterns that match the recognized text, pertinent data pertaining tothe known text patterns that match the recognized text from one or moredata sources; wherein displaying comprises: displaying, at the displayusing augmented reality, at least some of the pertinent data pertainingto the known text patterns to supplement information in the imagesacquired by the imaging device.
 4. A method according to claim 2,wherein analyzing the images, comprises: analyzing the images, at theprocessor, to find the matched objects that resemble a person; whereinperforming recognition on the matched objects, at the processor, torecognize the target components, comprises: performing facialrecognition on the matched objects that resemble the person, at theprocessor, to recognize facial features the person; wherein determiningcomprises: determining, at the processor, whether the recognizedfeatures match one or more known facial patterns, and generating facialrecognition results identifying a specific person having a known facialpattern that matches the recognized features; wherein retrievingcomprises: retrieving pertinent data pertaining to the specific personfrom the one or more data sources; wherein displaying comprises:displaying, at the display using augmented reality, at least some of thepertinent data pertaining to the specific person to supplementinformation in the images acquired by the imaging device.
 5. A methodaccording to claim 2, wherein analyzing the images, comprises: analyzingthe images, at the processor, to find matched objects that resemble anapparatus and identifiable information in the images; wherein performingrecognition on the matched objects, at the processor, to recognize thetarget components, comprises: performing recognition on the matchedobjects that resemble the apparatus and the identifiable information, atthe processor, to recognize the apparatus and the identifiableinformation; wherein determining comprises: determining, at theprocessor, whether the recognized apparatus matches any known apparatus,and generating recognition results identifying the known apparatus thatmatches the recognized apparatus as a specific apparatus; whereinretrieving comprises: retrieving, based on the identifiable information,supplemental information that relates to the specific apparatus from theone or more data sources; wherein displaying comprises: displaying, atthe display using augmented reality, at least some of the supplementalinformation that relates to the specific apparatus to supplementinformation in the images acquired by the imaging device.
 6. A methodaccording to claim 2, further comprising: specifying a desireddestination; wherein analyzing the images, comprises: analyzing theimages, at the processor, to find matched objects that resemblepotential landmarks in the images; wherein performing recognition on thematched objects, at the processor, to recognize the target components,comprises: performing recognition on the matched objects that resemblethe potential landmarks in the images, at the processor, to recognizelandmarks in the images; wherein determining comprises: determining, atthe processor, whether each of the recognized landmarks matches anyknown landmarks, and generating recognition results identifying specificlandmarks that match any known landmarks and location of each specificlandmark; wherein retrieving comprises: retrieving, based on locationsof each specific landmark and the desired destination, one or more pathsbetween the recognized landmarks and the desired destination from theone or more data sources; and retrieving, from the one or more datasources, supplemental information that is encountered along each paththat can be displayed while traversing that path; wherein displayingcomprises: displaying, at the display using augmented reality, a path asit is traversed along with at least some of the supplemental informationthat is encountered along that path while traversing that path.
 7. Amethod according to claim 2, wherein analyzing the images, comprises:analyzing the images, at the processor, to find matched objects that arepotentially indicative of a current location in the images; whereinperforming recognition on the matched objects, at the processor, torecognize the target components, comprises: performing recognition basedon the matched objects that are potentially indicative of currentlocation in the images, at the processor, to recognize target componentsof each object; wherein determining comprises: determining, at theprocessor, whether each of the recognized target components match anyknown patterns, and generating recognition results identifying specificobjects having target components determined to match any known patternsas being objects indicative of current location; further comprising:determining the current location based on the specific objects; andreceiving a selection of one or more filters to be applied forgenerating supplemental information; and wherein retrieving comprises:applying the one or more filters at the one or more data sources, andretrieving, from the one or more data sources, supplemental informationthat is to be displayed, wherein the supplemental information isdetermined based on the one or more filters; wherein displayingcomprises: displaying, at the display using augmented reality, thesupplemental information, wherein the supplemental information indicatesother hidden features associated with the current location that areobstructed from view and not visible.
 8. A method according to claim 2,wherein the one or more data sources comprise: a backend database; abackend server system; a cloud computing platform; targets identified bya search engine; and a social media platform or service.
 9. A computingsystem, comprising: a server system comprising: a memory comprisingprocessor-executable instructions; and a processor configured to executethe processor-executable instructions, wherein the processor-executableinstructions, when executed by the processor, are configurable to cause:analyzing images provided from a mobile communication device to find oneor more matched objects; determining whether recognized targetcomponents from the matched objects match one or more known patterns;and generating pertinent data pertaining to the known patterns tosupplement information in the images, wherein the pertinent data is tobe displayed at a display of the mobile communication device usingaugmented reality.
 10. A computing system according to claim 8, furthercomprising: a mobile communication device: a display and an imagingdevice configured to acquire the images; and wherein the server systemfurther comprises: a network interface configured to communicate withone or more data sources; and wherein the processor-executableinstructions, when executed by the processor, are configurable to cause:receiving the images acquired by the imaging device of the mobilecommunication device; performing recognition processing on the matchedobjects to recognize target components; and retrieving, based on theknown patterns, the pertinent data pertaining to the known patterns fromone or more data sources; and wherein the display of the mobilecommunication device is further configured to: display a user interfaceat the display, wherein the user interface is configured to present,using augmented reality, at least some of the pertinent data tosupplement the information in the images acquired by the imaging device.11. A computing system according to claim 10, wherein the matchedobjects comprise: matched objects that resemble text, and wherein theprocessor-executable instructions, when executed by the processor, arefurther configurable to cause: performing text recognition on thematched objects that resemble text to recognize text; determiningwhether the recognized text matches one or more known text patterns;generating text recognition results identifying the known text patternsthat match the recognized text; and retrieving, based on the known textpatterns that match the recognized text, pertinent data pertaining tothe known text patterns that match the recognized text from one or moredata sources.
 12. A computing system according to claim 10, wherein thematched objects comprise: matched objects that resemble a person, andwherein the processor-executable instructions, when executed by theprocessor, are further configurable to cause: performing facialrecognition on the matched objects that resemble the person to recognizefacial features the person; determining whether the recognized featuresmatch one or more known facial patterns; generating facial recognitionresults identifying a specific person having a known facial pattern thatmatches the recognized features; and retrieving pertinent datapertaining to the specific person from the one or more data sources; andwherein the user interface is configured to present, using augmentedreality, at least some of the pertinent data pertaining to the specificperson to supplement information in the images acquired by the imagingdevice.
 13. A computing system according to claim 10, wherein thematched objects comprise: matched objects that resemble an apparatus andidentifiable information in the images, and wherein theprocessor-executable instructions, when executed by the processor, arefurther configurable to cause: performing recognition on the matchedobjects that resemble the apparatus and the identifiable information torecognize the apparatus and the identifiable information; determiningwhether the recognized apparatus matches any known apparatus; generatingrecognition results identifying the known apparatus that matches therecognized apparatus as a specific apparatus; and retrieving, based onthe identifiable information, supplemental information that relates tothe specific apparatus from the one or more data sources, and whereinthe user interface is configured to present, using augmented reality, atleast some of the supplemental information that relates to the specificapparatus to supplement information in the images acquired by theimaging device.
 14. A computing system according to claim 10, whereinthe matched objects comprise: matched objects that resemble potentiallandmarks in the images, and wherein the processor-executableinstructions, when executed by the processor, are further configurableto cause: receiving an input that indicates a desired destination;performing recognition on the matched objects that resemble thepotential landmarks in the images to recognize landmarks in the images;determining whether each of the recognized landmarks matches any knownlandmarks; generating recognition results identifying specific landmarksthat match any known landmarks and location of each specific landmark;retrieving, based on locations of each specific landmark and the desireddestination, one or more paths between the recognized landmarks and thedesired destination from the one or more data sources; and retrieving,from the one or more data sources, supplemental information that isencountered along each path that can be displayed while traversing thatpath; and wherein the user interface is configured to present, usingaugmented reality, a path as it is traversed along with at least some ofthe supplemental information that is encountered along that path whiletraversing that path.
 15. A computing system according to claim 10,wherein the matched objects are potentially indicative of a currentlocation in the images, and wherein the processor-executableinstructions, when executed by the processor, are further configurableto cause: performing recognition based on the matched objects that arepotentially indicative of current location in the images to recognizetarget components of each object; determining whether each of therecognized target components match any known patterns; generatingrecognition results identifying specific objects having targetcomponents determined to match any known patterns as being objectsindicative of current location; determining the current location basedon the specific objects; receiving a selection of one or more filters tobe applied for generating supplemental information; applying the one ormore filters at the one or more data sources; and retrieving, from theone or more data sources, supplemental information that is to bedisplayed; and wherein the user interface is configured to present,using augmented reality, the supplemental information, wherein thesupplemental information is determined based on the one or more filtersand indicates other hidden features associated with the current locationthat are obstructed from view and not visible.
 16. A computing systemaccording to claim 10, wherein the processor-executable instructions,when executed by the processor, are further configurable to causefiltering the pertinent data retrieved from one or more data sourcesprior to displaying.
 17. A mobile communication device, comprising: animaging device configured to acquire images; a memory comprisingprocessor-executable instructions; and a processor configured to executethe processor-executable instructions, wherein the processor-executableinstructions, when executed by the processor, are configurable to cause:analyzing the images acquired by the imaging device to find one or morematched objects from one or more data sources; and determining whetherrecognized target components from the matched objects match one or moreknown patterns; and retrieving, pertinent data pertaining to the knownpatterns from the one or more data sources; and a display configured todisplay a user interface that is configured to present, using augmentedreality, at least some of the pertinent data to supplement informationin the images acquired by the imaging device.
 18. A mobile communicationdevice according to claim 17, further comprising: a network interfaceconfigured to communicate with one or more data sources; and wherein theprocessor-executable instructions, when executed by the processor, arefurther configurable to cause: performing recognition processing on thematched objects to recognize target components; and retrieving, based onthe known patterns, the pertinent data pertaining to the known patternsfrom one or more data sources.
 19. A mobile communication deviceaccording to claim 17, wherein the matched objects comprise: matchedobjects that resemble text, and wherein the processor-executableinstructions, when executed by the processor, are further configurableto cause: performing text recognition on the matched objects thatresemble text to recognize text; determining whether the recognized textmatches one or more known text patterns; generating text recognitionresults identifying the known text patterns that match the recognizedtext; and retrieving, based on the known text patterns that match therecognized text, pertinent data pertaining to the known text patternsthat match the recognized text from one or more data sources.
 20. Amobile communication device according to claim 17, wherein the matchedobjects comprise: matched objects that resemble a person, and whereinthe processor-executable instructions, when executed by the processor,are further configurable to cause: performing facial recognition on thematched objects that resemble the person to recognize facial featuresthe person; determining whether the recognized features match one ormore known facial patterns; generating facial recognition resultsidentifying a specific person having a known facial pattern that matchesthe recognized features; and retrieving pertinent data pertaining to thespecific person from the one or more data sources; and wherein the userinterface is configured to present, using augmented reality, at leastsome of the pertinent data pertaining to the specific person tosupplement information in the images acquired by the imaging device.